Auto discovery and auto scaling of services in software-defined network environment

ABSTRACT

Techniques are described for automatic discovery of two or more virtual service instances configured to apply a given service to a packet in a software-defined networking (SDN)/network functions virtualization (NFV) environment. Virtual service instances may be deployed as virtual entities hosted on one or more physical devices to offer individual services or chains of services from a service provider. The use of virtual service instances enables automatic scaling of the services on-demand. The techniques of this disclosure enable automatic discovery by a gateway network device of virtual service instances for a given service as load balancing entities. According to the techniques, the gateway network device automatically updates a load balancing group for the given service to include the discovered virtual service instances on which to load balance traffic for the service. In this way, the disclosed techniques provide auto-scaling and auto-discovery of services in an SDN/NFV environment.

TECHNICAL FIELD

The disclosure relates to computer networks and, more specifically, tonetwork services in computer networks.

BACKGROUND

A computer network is a collection of interconnected computing devicesthat can exchange data and share resources. In a packet-based network,the computing devices communicate data by dividing the data into smallblocks called packets, which are individually routed across the networkfrom a source device to a destination device. Certain devices, referredto as routers, within a computer network maintain routing informationthat describes routes through the network. A “route” can generally bedefined as a path between two locations on the network. Routers includea control plane that maintains the routing information, and a data planethat forwards received packets according to the routing information.

Network service providers provide services such as security, tunneling,virtual private networks, filtering, load-balancing, VoIP/Multimediaprocessing and various types of application proxies (HTTP, XML, WAP,etc.) to incoming packets. Service providers also providecontent-specific services designed to improve the quality of a user'sexperience, for example, video streaming and caching. The networkinfrastructure of a service provider network typically includes a vastcollection of access nodes, aggregation nodes and high-speed edgerouters interconnected by communication links. These network devicestypically execute various protocols and exchange signaling messages toanchor and manage subscriber sessions and communication flows associatedwith subscriber devices.

A software defined networking (SDN) controller and/or a networkfunctions virtualization (NFV) orchestrator may be included in thenetwork architecture to provide centralized control of the subscribersessions and communication flows within the service provider network. AnSDN architecture is often used to provide at least a degree ofseparation of the control plane and the data plane in network devices,and the abstraction of the control plane into a more modular and layeredarchitecture. An NFV architecture provides virtualization to removedependency on specialized hardware and to consolidate many differentnetwork equipment types onto industry standard high volume servers,switches and storage, which may be located in data centers, networknodes and in end user premises.

SUMMARY

In general, techniques are described for automatic discovery of two ormore virtual service instances configured to apply a given service to apacket in a software-defined networking (SDN)/network functionsvirtualization (NFV) environment. Virtual service instances may bedeployed as virtual machines (VMs) or other virtual entities hosted onone or more physical devices, e.g., servers, to offer individualservices or chains of services from a service provider. The use ofvirtual service instances enables automatic scaling (i.e., instantiatingor tearing down) of the services on-demand. The techniques of thisdisclosure enable automatic discovery by a gateway network device, e.g.,a router, of virtual service instances for a given service as loadbalancing entities. According to the techniques, the gateway networkdevice automatically updates a load balancing group for the givenservice to include the discovered virtual service instances on which toload balance traffic for the service. In this way, the disclosedtechniques provide auto-scaling and auto-discovery of services in anSDN/NFV environment.

In one example, this disclosure is directed to a method comprisingestablishing, by a SDN controller, a border gateway protocol (BGP) peersession with a gateway network device; establishing, by the SDNcontroller, two or more unique tunnels that each extends from thegateway network device to a respective one of a set of two or morevirtual service instances configured to apply a given service to apacket; calculating, by the SDN controller, BGP routes for the givenservice, wherein each of the BGP routes represents one of the uniquetunnels that extends from the gateway network device to the respectiveone of the virtual service instances; and advertising, by the SDNcontroller and to the gateway network device via the BGP peer session,the BGP routes for the given service as load balancing entities.

In another example, this disclosure is directed to a method comprisingestablishing, by a gateway network device, a BGP peer session with a SDNcontroller; receiving, by the gateway network device and from the SDNcontroller via the BGP peer session, advertisements of BGP routes for agiven service as load balancing entities, wherein each of the BGP routesrepresents one of a set of two or more unique tunnels that each extendsfrom the gateway network device to a respective one of a set of two ormore virtual service instances configured to apply the given service toa packet; maintaining, by the gateway network device, a load balancinggroup for the given service that includes each of the BGP routes for thegiven service; and selecting, by the gateway network device andaccording to a load balancing algorithm, one of the BGP routes from theload balancing group on which to forward the packet to the respectiveone of the virtual service instances for application of the givenservice.

In a further example, this disclosure is directed to a SDN controllercomprising a plurality of network interfaces; and a control unitcomprising a processor executing BGP to establish a BGP peer session onone of the network interfaces with a gateway network device, andestablish two or more unique tunnels that each extends from the gatewaynetwork device to a respective one of a set of two or more virtualservice instances configured to apply a given service to a packet, theprocessor configured to calculate BGP routes for the given service,wherein each of the BGP routes represents one of the unique tunnels thatextends from the gateway network device to the respective one of thevirtual service instances, and output the BGP routes for the givenservice to the one of the network interfaces for advertising as loadbalancing entities to the gateway network device via the BGP peersession.

In an additional example, this disclosure is directed to a gatewaynetwork device comprising a plurality of network interfaces; a controlunit comprising a processor executing BGP to establish a BGP peersession on one of the network interfaces with a SDN controller, theprocessor configured to receive advertisements of BGP routes for a givenservice as load balancing entities on the one of the network interfacesvia the BGP peer session with the SDN controller, wherein each of theBGP routes represents one of a set of two or more unique tunnels thateach extends from the gateway network device to a respective one of aset of two or more virtual service instances configured to apply thegiven service to a packet, and maintain a load balancing group for thegiven service that includes each of the BGP routes for the givenservice; and a forwarding circuit programmed with forwarding informationassociated with the BGP routes, the forwarding circuit configured toselect one of the BGP routes for the packet from the load balancinggroup according to a load balancing algorithm, wherein the forwardingcircuit outputs the packet to one of the network interfaces forforwarding to the respective one of the virtual service instances forapplication of the given service.

The details of one or more embodiments of this disclosure are set forthin the accompanying drawings and the description below. Other features,objects, and advantages will be apparent from the description anddrawings, and from the claims.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an example network systemoperating in accordance with techniques described herein.

FIG. 2 is a block diagram illustrating a portion of an example serviceprovider network operating in accordance with techniques describedherein.

FIG. 3 is a block diagram illustrating an example SDN controllerconfigured to implement the techniques described herein.

FIG. 4 is a block diagram illustrating an example router configured toimplement the techniques described herein.

FIG. 5 is a block diagram illustrating an example server that providesan operating environment for one or more service nodes.

FIG. 6 is a flowchart illustrating exemplary operation of a networkdevice, e.g., a router or gateway, and a SDN controller, in accordancewith the techniques described herein.

DETAILED DESCRIPTION

FIG. 1 is a block diagram illustrating an example network systemoperating in accordance with techniques described herein. The examplenetwork system of FIG. 1 includes a service provider network 2 thatoperates as a private network to provide packet-based network servicesto subscriber devices 16. That is, service provider network 2 providesauthentication and establishment of network access for subscriberdevices 16 such that subscriber devices 16 may begin exchanging datapackets with public network 12, which may be an internal or externalpacket-based network such as the Internet.

In the example of FIG. 1, service provider network 2 comprises an accessnetwork 6 that provides connectivity to public network 12 via serviceprovider core network 7 and gateway network device 8. Service providercore network 7 and public network 12 provide packet-based services thatare available for request and use by subscriber devices 16. As examples,core network 7 and/or public network 12 may provide, for example, bulkdata delivery, voice over Internet protocol (VoIP), Internet Protocoltelevision (IPTV), Short Messaging Service (SMS), Wireless ApplicationProtocol (WAP) service, or customer-specific application services.Public network 12 may comprise, for instance, a local area network(LAN), a wide area network (WAN), the Internet, a virtual LAN (VLAN), anenterprise LAN, a layer 3 virtual private network (VPN), an InternetProtocol (IP) intranet operated by the service provider that operatesaccess network 6, an enterprise IP network, or some combination thereof.In various embodiments, public network 12 is connected to a public WAN,the Internet, or to other networks. Public network 12 executes one ormore packet data protocols (PDPs), such as IP (IPv4 and/or IPv6) orPoint-to-Point Protocol (PPP), to enable packet-based transport ofpublic network 12 services.

Subscriber devices 16 connect to gateway 8 via access network 6 toreceive connectivity to subscriber services for applications hosted bysubscriber devices 16. A subscriber may represent, for instance, anenterprise, a residential subscriber, or a mobile subscriber. Subscriberdevices 16 may be, for example, personal computers, laptop computers orother types of computing device associated with subscribers. Inaddition, subscriber devices 16 may comprise mobile devices that accessthe data services of service provider network 2 via radio access network(RAN) 4. Example mobile subscriber devices 16 include mobile telephones,laptop or desktop computers having, e.g., a 3G wireless card,wireless-capable netbooks, video game devices, pagers, smart phones,personal data assistants (PDAs) or the like. Each of subscriber devices16 may run a variety of software applications, such as word processingand other office support software, web browsing software, software tosupport voice calls, video games, videoconferencing, and email, amongothers. Subscriber devices 16 connect to access network 6 via accesslinks 5 that comprise wired and/or wireless communication links. Theterm “communication link,” as used herein, comprises any form oftransport medium, wired or wireless, and can include intermediate nodessuch as network devices. Each of access links 5 may comprise, forinstance, aspects of an asymmetric DSL network, WiMAX, a T-1 line, anIntegrated Service Digital Network (ISDN), wired Ethernet, or a cellularradio link.

A network service provider operates, or in some cases leases, elementsof access network 6 to provide packet transport between subscriberdevices 16 and gateway 8. Access network 6 represents a network thataggregates data traffic from one or more subscriber devices 16 fortransport to/from service provider core network 7 of the serviceprovider. Access network 6 includes network nodes that executecommunication protocols to transport control and user data to facilitatecommunication between subscriber devices 16 and gateway 8. Accessnetwork 6 may include a broadband access network, a wireless LAN, apublic switched telephone network (PSTN), or other type of accessnetwork, and may include or otherwise provide connectivity for cellularaccess networks, such as RAN 4. Examples of RAN 4 include networksconforming to a Universal Mobile Telecommunications System (UMTS)architecture, an evolution of UMTS referred to as Long Term Evolution(LTE), mobile IP standardized by the Internet Engineering Task Force(IETF), as well as other standards proposed by the 3^(rd) GenerationPartnership Project (3GPP), 3^(rd) Generation Partnership Project 2(3GGP/2) and the Worldwide Interoperability for Microwave Access (WiMAX)forum.

Service provider core network 7 offers packet-based connectivity tosubscriber devices 16 attached to access network 6 for accessing publicnetwork 12. Core network 7 may represent a public network that is ownedand operated by a service provider to interconnect a plurality ofnetworks, which may include access network 6. Core network 7 mayimplement Multi-Protocol Label Switching (MPLS) forwarding and, in suchinstances, may be referred to as an MPLS network or MPLS backbone. Insome instances, core network 7 represents a plurality of interconnectedautonomous systems, such as the Internet, that offers services from oneor more service providers. Public network 12 may represent an edgenetwork coupled to core network 7, e.g., by a customer edge device suchas customer edge switch or router. Public network 12 may include a datacenter.

In examples of service provider network 2 that include awireline/broadband access network as access network 6, gateway 8 mayrepresent a Broadband Network Gateway (BNG), Broadband Remote AccessServer (BRAS), MPLS Provider Edge (PE) router, core router or gateway,or a Cable Modem Termination System (CMTS), for instance. In examples ofservice provider network 2 that include a cellular access network asaccess network 6, gateway 8 may represent a mobile gateway, for example,a Gateway General Packet Radio Service (GPRS) Serving Node (GGSN), anAccess Gateway (aGW), or a Packet Data Network (PDN) Gateway (PGW). Inother examples, the functionality described with respect to gateway 8may be implemented in a switch, service card or other network element orcomponent.

A network service provider that administers at least parts of serviceprovider network 2 typically offers services to subscribers associatedwith devices, e.g., subscriber devices 16, that access service providernetwork 2. Services offered may include, for example, traditionalInternet access, Voice-over-Internet Protocol (VoIP), video andmultimedia services, and security services. As described above withrespect to access network 6, core network 7 may support multiple typesof access network infrastructures that connect to service providernetwork access gateways to provide access to the offered services. Insome instances, the network system illustrated in FIG. 1 may includesubscriber devices 16 that attach to multiple different access networks6 having varying architectures.

In general, any one or more of subscriber devices 16 may requestauthorization and data services by sending a session request to gateway8. In turn, gateway 8 typically accesses Authentication, Authorizationand Accounting (AAA) server 11 to authenticate the subscriber devicerequesting network access. Once authenticated, any of subscriber devices16 may send subscriber data traffic toward service provider core network7 in order to access and receive services provided by public network 12,and such packets traverse gateway 8 as part of at least one packet flow.Flows 26 illustrated in FIG. 1 represent one or more upstream packetflows from any one or more subscriber devices 16 and directed to publicnetwork 12. The term “packet flow,” “traffic flow,” or simply “flow”refers to a set of packets originating from a particular source deviceand sent to a particular destination device. A single flow of packets,in either the upstream (sourced by one of subscriber devices 16) ordownstream (destined for one of subscriber devices 16) direction, may beidentified by the 5-tuple: <source network address, destination networkaddress, source port, destination port, protocol>, for example. This5-tuple generally identifies a packet flow to which a received packetcorresponds. An n-tuple refers to any n items drawn from the 5-tuple.For example, a 2-tuple for a packet may refer to the combination of<source network address, destination network address> or <source networkaddress, source port> for the packet. Moreover, a subscriber device 16may originate multiple packet flows upon authentication to serviceprovider network 2 and establish a communication session for receivingdata services.

As described herein, service provider network 2 includes a servicescomplex 9 having a cluster of service nodes 10A-10N (collectively“service nodes 10”) that provide an execution environment for thenetwork services. That is, each of service nodes 10 applies one or moreservices. As examples, service nodes 10 may apply firewall and securityservices, carrier grade network address translation (CG-NAT), mediaoptimization (voice/video), IPSec/VPN services, deep packet inspection(DPI), HTTP filtering, counting, accounting, charging, and loadbalancing of packet flows or other types of services applied to networktraffic. In this way, each of service nodes 10 represents a serviceinstance.

Although illustrated as part of a services complex 9, which mayrepresent a data center, service nodes 10 may, for instance, be coupledby one or more switches or virtual switches of core network 7. In oneexample, each of service nodes 10 may run as virtual machines (VMs) orother virtual entities in a virtual computing environment. Moreover, thecomputing environment may comprise a scalable cluster of physicalnetwork devices, such as processor-based servers. As another example,service nodes 10 may comprise a combination of general purpose computingdevices and special purpose appliances. Virtual service instancesprovided by service nodes 10 can scale vertically, just as in a moderndata center, through the allocation of virtualized memory, processorutilization, storage and network policies, as well as horizontally byadding additional load-balanced virtual machines.

As shown in FIG. 1, gateway 8 steers individual subscriber packet flows26 through defined sets of services provided by service nodes 10. Thatis, each subscriber packet flow may be forwarded through a particularordered combination of services provided by service nodes 10, eachordered set being referred to herein as a “service chain,” and eachservice chain providing a composite network service represented by theorder and combination of individual network services (virtualizednetwork functions) provided by the service nodes of the service chain.In the example of FIG. 1, one or more subscriber packet flows 26 aredirected along a first service chain 28A and, therefore, receiveservices applied by service nodes 10A, 10B and 10N, in that order.Similarly, one or more subscriber packet flows 26 are directed along asecond service chain 28B and, therefore, receive services applied byservice nodes 10C, 10B and 10N.

In this way, subscriber flows 26 may be processed by service nodes 10 asthe packets flow between access network 6 and public network 12according to service chains configured by the service provider. In theillustrated example, service chain 28A identifies the ordered set ofnodes 10A, 10B, and 10N according to the listed ordering. Service chain28B identifies the ordered set of nodes 10C, 10B and 10N. Accordingly,packet flows 26 processed according to service chain 28A follow aservice path that traverses nodes 10A, 10B, and finally node 10N as theterminal node for the service chain 28A. A particular service node maysupport multiple service chains. In this example, service node 10Bsupports service chains 28A, 28B. Once processed at a terminal node ofthe service chain, i.e., the last node to apply services to packetsflowing along a particular service path, the terminal node may directthe traffic back to gateway 8 for further processing and/or forwardingto public network 12. For example, traffic engineered service paths maystart and terminate with gateway 8.

Whereas a “service chain” defines one or more services to be applied ina particular order to provide a composite service for application topacket flows bound to the service chain, a “service tunnel” or “servicepath” refers to a logical and/or physical path taken by packet flowsprocessed by a service chain along with the forwarding state forforwarding packet flows according to the service chain ordering. Eachservice chain may be associated with a respective service tunnel, andpacket flows associated with each subscriber device 16 flow alongservice tunnels in accordance with a service profile associated with therespective subscriber. The arrows denoted as service chains 28A, 28Billustrate respective paths taken by packet flows mapped to the servicechains 28A or 28B. For example, a given subscriber device 16 may beassociated with a particular service profile, which in turn is mapped toa service tunnel associated with service chain 28A. Similarly, anothersubscriber device 16 may be associated with a different service profile,which in turn is mapped to a service tunnel associated with servicechain 28B. Gateway 8, after authenticating and establishing accesssessions for the subscribers, directs packet flows for subscriberdevices 16 along the appropriate service tunnels, thereby causingservice complex 9 to apply the requisite ordered services for the givensubscriber device.

Service nodes 10 may implement service chains 28A, 28B using internallyconfigured forwarding state that directs packets of the packet flow longthe service chains 28A, 28B for processing according to the identifiedset of service nodes 10. Such forwarding state may specify tunnelinterfaces for tunneling between service nodes 10 using network tunnelssuch as Internet Protocol (IP) or Generic Route Encapsulation (GRE)tunnels, or by using Virtual Local Area Networks (VLANs), MultiprotocolLabel Switching (MPLS) techniques, and so forth. In some instances, realor virtual switches, routers or other network elements that interconnectservice nodes 10 may be configured to direct packet flows to servicenodes 10 according to service chains 28A, 28B.

In FIG. 1, software-defined networking (SDN) controller 19 provides ahigh-level controller for configuring and managing routing and switchinginfrastructure of service provider network 2 (e.g., gateway 8, corenetwork 7 and service nodes 10). In some instances, SDN controller 19manages deployment of virtual machines (VMs) or other virtual entitieswithin the operating environment of services complex 9. For example, SDNcontroller 19 may establish one or more of service nodes 10 as a virtualservice instance configured to apply a given service to packets. In thisexample, service provider network 2 may comprise a SDN/network functionsvirtualization (NFV) environment.

In addition, SDN controller 19 may interact with gateway 8 to specifyservice chain 28A, 28B information. For example, the service chaininformation provided by SDN controller 19 may specify any combinationand ordering of services provided by service nodes 10, trafficengineering information (e.g., labels or next hops) for tunneling orotherwise transporting (e.g., MPLS or IP tunnels) packet flows alongservice paths, rate limits, Type of Service (TOS) markings or packetclassifiers that specify criteria for matching packet flows to aparticular service chain 28A, 28B. Further example details of an SDNcontroller are described in PCT International Patent ApplicationPCT/US13/44378, filed Jun. 5, 2013, the entire contents of which areincorporated herein by reference.

Based on the specification of the service chain 28A, 28B information,SDN controller 19 may also establish unique tunnels for transportingpackets of packet flows 26 between gateway 8 and service nodes 10, andfor transporting packet between service nodes 10 within services complex9. For example, SDN controller 19 may perform path computation and issuepath setup messages and/or distribute labels or other header informationto be used at each hop along each service path. Example details of anSDN controller capable of performing path computation and distributinglabels to network elements are described in U.S. patent application Ser.No. 13/842,453, filed Mar. 15, 2013, the entire contents of which areincorporated herein by reference.

As discussed above, service nodes 10 may comprise virtual serviceinstances deployed as VMs or other virtual entities hosted on one ormore physical devices, e.g., servers, to offer individual services orchains of services. The use of virtual service instances enablesautomatic scaling of the services on-demand. In one example, SDNcontroller 19 may establish or instantiate additional virtual serviceinstances within services complex 9 for a given service based on anincreased needed for the given service by packets of packet flows 26received by gateway 8. In another example, SDN controller 19 may removeor tear down virtual service instances from services complex 9 for agiven service based on a decreased needed for the given service bypackets of packet flows 26 received by gateway 8.

To achieve service chaining with high scale and throughput, gateway 8may be used to anchor packets of packet flows 26 to the services offeredby service nodes 10 in both a forward and a reverse direction. Inaddition, gateway 8 may be used to perform load balancing of packetflows 26 to first or front-end services in the service chains. In somescenarios, one virtual service instance or VM per service may not beenough to support the service needs of packet flows 26. In this case,complete functionality for a given service may span across multiple VMshosted on one or more physical servers. Hence, a need for load balancingacross virtual service instances may arise at gateway 8. For example, insituations where two or more of service nodes 10, e.g., service nodes10A and 10C, offer the same service, gateway 8 may include service nodes10A, 10C in a load balancing group and load balance packets assigned tothat service across tunnels to service nodes 10A, 10C.

Conventionally, a load balancing group for a given service is configuredmanually to include the service-providing entities, e.g., real orvirtual service instances of services nodes 10, operating at a point intime. The load balancing group may be manually updated periodically toreflect changes in the service-providing entities. In the case ofvirtual service instances, however, manual configuration of the loadbalancing group for the given service may be unable to accurately trackthe on-demand scaling (i.e., instantiating and/or tearing down) of thevirtual service instances for the given service. Since the virtualservice instances can be dynamically added, deleted, or migrated onphysical network devices, and the services can go down and come up dueto scheduled maintenance, it may be desirable for gateway 8 to be ableto automatically discover the virtual service instances established fora given service.

According to the techniques of this disclosure, gateway 8 and SDNcontroller 19 are configured to enable automatic discovery of two ormore virtual service instances, e.g., two or more of service nodes 10,configured to apply a given service to packets of packet flows 26.According to the techniques, SDN controller 19 and gateway 8 establish aborder gateway protocol (BGP) peer session 18 on which SDN controller 19advertises BGP routes for the service. The advertised BGP routesrepresent unique tunnels established by SDN controller 19 that eachextends from gateway 8 to a respective one of the virtual serviceinstances of service nodes 10 for the given service. Gateway 8 discoversthe virtual service endpoints of the advertised BGP routes as realservices, and automatically updates a load balancing group for the givenservice to include the BGP routes. In this way, gateway 8 and SDNcontroller 19 support both auto-scaling and auto-discovery of servicesin an SDN/NFV environment, and a load balancer on gateway 8 can adjustalong with the dynamic services in the SDN/NFV environment.

Upon receipt of packets of packet flows 26 assigned to the given serviceor a service chain with the given service as the front-end service, aload balancer of gateway 8 selects one of the BGP routes from the loadbalancing group for the given service. Gateway 8 may modify the packetsof packet flows 26 as gateway 8 steers the packets into the appropriateservice or service chain. For example, gateway 8 may prepend to eachpacket a header and/or label (e.g., an additional IP header and/or MPLSlabel) to form a “tunnel packet” in which the packet is encapsulated asa payload. Gateway 8 may select the particular header based on theparticular subscriber and the service chain to which the particular typeof packet flow for the particular subscriber is assigned. In accordancewith the techniques of this disclosure, gateway 8 may further select theparticular header and/or label based on the one of the BGP routesselected from the load balancing group for the front-end service of theservice chain. Gateway 8 then forwards the encapsulated packets based onthe header and/or label for the selected BGP route to the virtualservice endpoint for application of the given service to the packets.

In some example implementations, the functionality and operationsdescribed herein in reference to SDN controller 19 may be implemented bya combination of devices or components. For example, in one exampleimplementation, the functionality and operations described herein inreference to SDN controller 19 may be implemented by an orchestrationengine operating in concert with an SDN controller that managesunderlying network state for routing and switching components so as toprovide the requested network and network services. As such, SDNcontroller 19 of FIG. 1 may represent an orchestration engine thatmanages networking services and an underlying SDN controller thatimplements high-level requests from the orchestration engine byconfiguring physical switches, e.g. TOR switches, chassis switches, orswitch fabric of the service provider core; physical routers; physicalservice nodes such as firewalls and load balancers and the like.Although not shown in FIG. 1, any reference to the term SDN controllerherein is understood to encompass implementations in which anorchestration engine is used as part of or in conjunction with the SDNcontroller. Further example details of an orchestration engine aredescribed in ETSI GS NFV-MAN 001, Network Functions Virtualisation (NFV)Management and Orchestration, dated December 2014, the entire contentsof which are incorporated herein by reference.

FIG. 2 is a block diagram illustrating a portion of an example serviceprovider network operating in accordance with techniques describedherein. The service provider network illustrated in FIG. 2 includes anSDN controller 32, a router 34, a first physical server-1 40, and asecond physical server-2 42. The SDN controller 32 may operatesubstantially similar to SDN controller 19 from FIG. 1. Router 34 mayoperate substantially similar to gateway network device 8 from FIG. 1.As described in more detail below, server-1 40 hosts virtual machines(VMs) 44, 45 and server-2 42 hosts VMs 46, 47 that are each configuredas virtual service instances. The VMs hosted by server-1 40 and server-242 may operate substantially similar to service nodes 10 from FIG. 1.

SDN controller 32 operates as a network service controller andestablishes two or more virtual service instances of a given service onat least one physical network device, e.g., service-1 40 and/or server-242. The service provider network illustrated in FIG. 2 offers a “green”service and a “red” service. Each of the green service and the redservice may be any of a firewall and security service, CG-NAT service,media optimization service, IPSec/VPN service, DPI service, HTTPfiltering service, or other types of services applied to networktraffic.

In the illustrated example of FIG. 2, server-1 40 executes a virtualrouter (VR) 41 that hosts a first virtual service instance VM-S11 44 anda second virtual service instance VM-S12 45 for the green service. Inthis example, VR 41 may be configured with a green service-specificroute target (RT), e.g., 65412:1. In addition, server-2 42 executes VR43 that hosts a third virtual service instance VM-S21 46 for the greenservice and a first virtual service instance VM-22 47 for the redservice. In this example, VR 43 may be configured with the greenservice-specific RT, e.g., 65412:1, and with a red service-specific RT,e.g., 75412:1.

SDN controller 32 also establishes unique tunnels that each extends fromrouter 34 to a respective one of the virtual service instancesconfigured to apply the given service to packet flows. FIG. 2illustrates a first unique tunnel that extends from router 34 to firstvirtual service instance VM-S11 44 of the green service being identifiedby a tunnel address of GRE tunnel 50 with label L101 assigned to VM-S1144, and a second unique tunnel that extends from router 34 to secondvirtual service instance VM-S12 45 of the green service being identifiedby the tunnel address of GRE tunnel 50 with label L102 assigned toVM-S12 45. In one example, the tunnel address of GRE tunnel 50 may begr-1/0/0.32270. FIG. 2 illustrates a third unique tunnel that extendsfrom router 34 to third virtual service instance VM-S21 46 of the greenservice being identified by a tunnel address of GRE tunnel 52 with labelL103 assigned to VM-S21 46. In one example, the tunnel address of GREtunnel 52 may be gr-1/1/0.32271. In addition, FIG. 2 illustrates afourth unique tunnel that extends from router 34 to first virtualservice instance VM-S22 47 of the red service being identified by atunnel address of GRE tunnel 53 with label L104 assigned to VM-S22 47.In one example, the tunnel address of GRE tunnel 53 may begr-1/2/0.32272.

SDN controller 32 and router 34 establish a BGP peer session 33 overwhich SDN controller 32 and router 34 exchange BGP messages, such asrouting information. SDN controller 32 calculates BGP routes thatrepresent the unique tunnels between router 34 and the virtual serviceinstances for the green service and the red service. SDN controller 32then advertises the BGP routes for each of the green service and the redservice to router 34 via BGP peer session 33. For example, for the greenservice, SDN controller 32 may calculate and advertise an ECMP next hoproute set for the green service including a first member route to reachVM-S11 44, a second member route to reach VM-S12 45, and a third memberroute to reach VM-S21 46. As another example, for the red service, SDNcontroller 32 may calculate and advertise an ECMP next hop route set forthe red service including a first member route to reach VM-S22 47.

Within router 34, the BGP and routing-instances related configurationsare setup appropriately to listen to BGP routes from SDN controller 32.For example, router 34 includes virtual routing and forwarding (VRF)route distinguisher (RD) attributes and VRF route target (RT) attributesthat may be configured to listen for ECMP next hops advertised by SDNcontroller 32 via BGP peer session 33. Whenever SDN controller 32configures one or more virtual service instances (e.g., VMs) on physicalnetwork devices, SDN controller 32 communicates the BGP routes to reachthe virtual service instances to router 34 via BGP peer session 33. Inthis way, SDN controller 32 may set up an ECMP next hop route set foreach service in router 34 with each member route of the ECMP next hoproute set representing one of the unique tunnels that extends to one ofthe virtual service instances of the service.

A load balancer of router 34 considers each of the virtual serviceendpoints of the unique tunnels represented by the BGP routes to be realservices for the purpose of load balancing. According to the disclosedtechniques, router 34 maintains a load balancing group for each of thegreen service and the red service that includes the BGP routes for thegiven service as load balancing entities. In the illustrated example ofFIG. 2, router 34 includes a VRF-Green table 36 for the green service,and a VRF-Green routing instance 37 that includes the greenservice-specific RT. Similarly, router 34 includes a VRF-Red table 38for the red service, and a VRF-Red routing instance 39 that includes thered service-specific RT. In the example of FIG. 2, router 34 maintainsthe load balancing group for the green service in VRF-Green table 36,and maintains the load balancing group for the red service in VRF-Redtable 38.

For each of the BGP routes included in the load balancing group for thegiven service, the VRF table for the service includes a tunnelidentifier for the unique tunnel that extends to the virtual serviceinstance represented by the respective one of the BGP routes. Forexample, VRF-green table 36 may include a first tunnel identifier forthe first unique tunnel that extends to first virtual service instanceVM-S 11 44 comprising a tunnel address of GRE tunnel 50 with label L101assigned to VM-S11 44. VRF-green 36 may also include a second tunnelidentifier for the second unique tunnel that extends to second virtualservice instanceVM-S12 45 comprising a tunnel address of GRE tunnel 50with label L102 assigned to VM-S12 45. VRF-green 36 may further includea third tunnel identifier for the third unique tunnel that extends tothird virtual service instance VM-S21 46 comprising a tunnel address ofGRE tunnel 52 with label L103 assigned to VM-S21 46. As another example,VRF-red table 38 may include a first tunnel identifier for the firstunique tunnel that extends to first virtual service instance VM-S22 47comprising a tunnel address of GRE tunnel 53 with label L104 assigned toVM-S22 47.

In one example, VRF-Green table 36 may include the following ECMP nexthop route set for the green service.

VRF-Green (route learnt via BGP)=0.0.0.0/0

ECMP {(gr-1/0/0.32270, L101), (gr-1/0/0.32270, L102), (gr-1/1/0.32271,L103)}

In addition, VRF-Green routing instance 37 may be configured to importthe green service-specific RT of VR 41 and VR 43: Import RT=65412:1.

In another example, VRF-Red table 38 may include the following ECMP nexthop route set for the red service.

VRF-Red (route learnt via BGP)=0.0.0.0/0 ECMP {(gr-1/2/0.32272, L104)}

In addition, VRF-Red routing instance 39 may be configured to import thered service-specific RT of VR 43: Import RT=75412:1.

Once the VRF tables and load balancing groups are configured in router34, router 34 may perform load balancing across the unique tunnels thatextend to virtual service endpoints for a given service. For example, inresponse to receiving a packet having a source IP address of 41.1.1.1and a destination IP address of 100.1.1.1, router 34 may first determinewhich service or service chain is assigned to the packet. In the casewhere the assigned service, or the front-end service of the assignedservice chain, is the green service, router 34 accesses the loadbalancing group in VRF-Green 36. The load balancer of router 34 performsload balancing based on a load balancing algorithm to select one of theBGP routes from the load balancing group on which to forward the packetto reach one of the virtual service instances VM-S11 44, VM-S12 45, orVM-S21 46 of the green service.

Router 34 may then encapsulate the received packet with the tunnelidentifier for the unique tunnel that extends to the virtual serviceinstance represented by the selected one of the BGP routes. For example,in the case where the load balancer of router 34 selects the BGP routethat represents the second unique tunnel that extends to the secondvirtual service instance VM-S12 45, router 34 may encapsulate thereceived packet with the second tunnel identifier in VRF-Green 36, whichincludes a tunnel address of GRE tunnel 50 with label L102 assigned toVM-S12 45, e.g., gr-1/0/0.32270, L102. Router 34 then forwards theencapsulated packet according to the tunnel identifier for the selectedBGP route to the virtual service instance for application of the greenservice to the packet.

In response to receiving the encapsulated packet via GRE tunnel 50, VR41 processes and removes the GRE tunnel address, e.g., gr-1/0/0.32270,of the tunnel identifier used to reach server-1 40 from the encapsulatedpacket. VR 41 then reads and removes the label, e.g., L102, assigned tothe second virtual service instance VM-S12 45. Based on the label, VR 41forwards the decapsulated, original packet, e.g., having the source IPaddress of 41.1.1.1 and the destination IP address of 100.1.1.1, onto anappropriate tap interface of VM-S12 45. VM-S12 45 then applies therequired service on the original packet.

One or more advantages of the disclosed techniques are described below.The techniques do not require any manual configuration as SDN controller32 and router 34 automatically discover the real services dynamicallyand support auto scaling. The techniques use BGP to learn the realservices from SDN controller 32. The techniques enable the load balancerof router 34 to gracefully handle the automatic scaling events of thereal services scaling up and down, and scaling in and out. Thetechniques require no explicit health monitoring to the backend realservices. Instead, BGP ECMP next hop updates may indicate which realservices are down or deleted. The techniques solve the use case of loadbalancing in a SDN/NFV environment. The techniques enable more capacityto be dynamically added in SDN/NFV environment whether or not thevirtual service instances are hosted on the same physical server ordifferent servers. The techniques do not use domain name system (DNS)and, thus, do not require any DNS setup, DNS servers, and DNSconfiguration. Instead, according to the techniques of this disclosure,auto-scaling and auto-discovery of services is fully automatic via BGP.

FIG. 3 is a block diagram illustrating an example SDN controller 60configured to implement the techniques described herein. SDN controller60 may operate as a network services controller for a service providernetwork. In some examples, SDN controller 60 may operate substantiallysimilar to SDN controller 19 from FIG. 1 and/or SDN controller 32 fromFIG. 2.

In the illustrated example of FIG. 3, SDN controller 60 includes ananalytics unit 61, a configuration unit 62, and a control unit 64coupled to a network interface 66 to exchange packets with networkdevices by inbound link 67 and outbound link 68. In some examples,analytics unit 61, configuration unit 62, and/or control unit 64 may beimplemented as one or more processes executing on one or more virtualmachines of one or more physical computing devices. That is, whilegenerally illustrated and described as executing on a single SDNcontroller 60, aspects of these units may be delegated to othercomputing devices.

Each of analytics unit 61, configuration unit 62, and control unit 64may include one or more processors (not shown) that execute softwareinstructions, such as those used to define a software or computerprogram, stored to a computer-readable storage medium (not shown), suchas non-transitory computer-readable mediums including a storage device(e.g., a disk drive, or an optical drive) or a memory (such as Flashmemory or RAM) or any other type of volatile or non-volatile memory,that stores instructions to cause the one or more processors to performthe techniques described herein. Alternatively, or additionally, each ofanalytics unit 61, configuration unit 62, and control unit 64 maycomprise dedicated hardware, such as one or more integrated circuits,one or more application-specific integrated circuits (ASICs), one ormore Application Specific Special Processors (ASSPs), one or more FPGAs,or any combination of one or more of the foregoing examples of dedicatedhardware, for performing the techniques described herein. Thearchitecture of SDN controller 60 illustrated in FIG. 3 is shown forexample purposes only and should not be limited to this architecture. Inother examples, SDN controller 60 may be configured in a variety ofways.

Analytics unit 61 and configuration unit 62 may comprise a managementlayer of SDN controller 60, whereas control unit 64 may comprise acontrol layer of SDN controller 60. Analytics unit 61 may captureinformation from physical and/or virtual network elements within theservice provider network, e.g., gateway 8 and service nodes 10 withinservice provider network 2 from FIG. 1, and analyze the information foruse in managing the network services offered by the service provider.The information may include statistics, logs, events, and errors.

Configuration unit 62 stores configuration information for the networkelements within the service provider network. In some examples, theconfiguration information comprises a virtual network configuration.Configuration unit 62 may translate a high-level data model of theintended virtual network configuration to a lower-level data model foruse in interacting with the network elements.

Control unit 64 of SDN controller 60 implements a centralized controlplane for the service provider network that is responsible formaintaining a constantly changing network state. Control unit 64interacts with the network elements within the service provider networkto maintain a consistent network state across all of the networkelements. Control unit 64 provides an operating environment for acommand line interface daemon 75 (“CLI 75”) that provides an interfaceby which an administrator or other management entity may modify theconfiguration of SDN controller 60 using text-based commands. Controlunit 64 also provides an operating environment for several protocols 70,including Border Gateway Protocol (BGP) 72 and Extensible Messaging andPresence Protocol (XMPP) 74 illustrated in FIG. 3.

In some examples, control unit 64 uses XMPP 74 to communicate withnetwork elements within the service provider network, such as gateway 8or service nodes 10 within service provider network 2 from FIG. 1, by anXMPP interface (not shown). Virtual network route data, statisticscollection, logs, and configuration information may be sent asextensible markup language (XML) documents in accordance with XMPP 74for communication between SDN controller 60 and the network elements.Control unit 64 may also use XMPP 74 to communicate with one or both ofanalytics unit 61 and configuration unit 62 of SDN controller 60.Control unit 64 executes BGP 72 to establish BGP peer sessions with BGPspeakers and BGP listeners within the service provider network toexchange routing information. For example, control unit 64 may use BGP72 to establish a BGP peer session with a gateway network device, e.g.,BGP peer session 18 with gateway 8 from FIG. 1 or BGP peer session 33with router 34 from FIG. 2.

Control unit 64 includes an auto-scaling unit 76 configured tocommunicate with physical network devices, e.g., servers, using XMPP 74to establish or remove virtual network service instances in response tofluctuating demands for different services for packet flows in theservice provider network. For example, auto-scaling unit 74 mayestablish a set of two or more virtual service instances of a givenservice as two or more virtual machines (VMs) or other virtual entitiesrunning on at least one physical network device, such as a server.Auto-scaling unit 76 may also be configured to use BGP 72 to establishunique tunnels that each extends from a gateway network device in theservice provider network to a respective one of the set of instantiatedvirtual service instances of the given service. As discussed above withrespect to FIG. 2, each of the unique tunnels may have a tunnelidentifier that includes a tunnel address of a GRE tunnel that extendsfrom the gateway network device to the physical network device, and alabel assigned to one of the virtual service instances of the givenservice running on the physical network device.

In accordance with the disclosed techniques, control unit 64 alsoincludes an auto-discovery unit 76 configured to calculate BGP routesthat represent the unique tunnels between the gateway network device andthe virtual service instances of the given service. Auto-discovery unit76 is further configured to advertise the BGP routes for the givenservice to the gateway network device via a BGP peer session with thegateway network device as load balancing entities. For example,auto-discovery unit 76 may advertise an ECMP next hop route set for thegiven service that includes two or more member routes to reach the twoor more virtual service instances of the given service. Advertising theBGP routes with virtual service endpoints as load balancing entities, asdescribed herein, enables the gateway network device to automaticallydiscover the virtual service endpoints and perform load balancing on theBGP routes to forward packet flows to the given service.

FIG. 4 is a block diagram illustrating an example router 80 configuredto implement the techniques described herein. Router 80 may comprise anyrouter in a network having a SDN/NFV architecture, such as serviceprovider network 2 from FIG. 1. For example, router 80 may comprise agateway, an edge router, a core router or other device illustrated inFIG. 1 that directs packet flows to a service complex for application ofnetwork services.

In the example of FIG. 4, router 80 includes control unit 82 in whichrouting engine 86 provides control plane functionality for router 80.Router 80 also includes a plurality of packet-forwarding engines114A-114N (“PFEs 114”) and a switch fabric 118 that collectively providea data plane for forwarding network traffic. PFEs 114 receive and senddata packets via interface cards 112 (“IFCs 112”). In other embodiments,each of PFEs 114 may comprise more or fewer IFCs. Although not shown,PFEs 114 may each comprise a central processing unit (CPU) and a memory.In the illustrated example, routing engine 86 is connected to each ofPFEs 114 by a dedicated internal communication link 120. For example,dedicated link 120 may comprise a Gigabit Ethernet connection. Switchfabric 118 provides a high-speed interconnect for forwarding incomingdata packets between PFEs 114 for transmission over a network.

Control unit 82 provides an operating environment for routing engine 86and may be implemented solely in software, or hardware, or may beimplemented as a combination of software, hardware or firmware. Forexample, control unit 82 may include one or more processors that executesoftware instructions. In that case, routing engine 86 may includevarious software modules or daemons (e.g., one or more routing protocolprocesses, user interfaces, and the like), and control unit 82 mayinclude a computer-readable storage medium, such as computer memory orhard disk, for storing executable instructions.

Routing engine 86 provides an operating environment for a command lineinterface daemon 90 (“CLI 90”) that provides an interface by which anadministrator or other management entity may modify the configuration ofrouter 80 using text-based commands. Simple Network Management Protocoldaemon 91 (“SNMP 91”) comprises an SNMP agent that receives SNMPcommands from a management entity to set and retrieve configuration andmanagement information for router 80. Using CLI 90 and SNMP 91,management entities may enable/disable and configure services, installroutes, enable/disable and configure rate limiters, and configureinterfaces, for example.

Routing engine 86 also provides an operating environment for executionof various protocols 88 that may comprise software processes havinginstructions executed by a computing environment. Routing engine 86executes a routing protocol daemon (RPD) 92 that uses one or morerouting protocols, such as BGP 96 and internal gateway protocol (IGP)98, to maintain routing information in the form of routing informationbase (RIB) 104 that describes a topology of a network, and derives aforwarding information base (FIB) 106 in accordance with the routinginformation. In general, the routing information represents the overalltopology of the service provider network. RPD 92 interacts with kernel100 (e.g., by way of application programming interface (API) calls) toupdate RIB 104 based on routing protocol messages received by router 80.RIB 104 may include information defining a topology of the network,including one or more routing tables and/or link-state databases.Typically, the routing information defines routes (i.e., series of nexthops) through a network to destinations/prefixes within the networklearned via a distance-vector routing protocol (e.g., BGP 96) or definesthe network topology with interconnected links learned using a linkstate routing protocol (e.g., intermediate-system to intermediate-system(IS-IS) or open shortest path first (OSPF)) of IGP 98.

Routing engine 86 also executes one or more traffic engineeringprotocols to establish tunnels for forwarding subscriber packets throughthe ordered set of service nodes associated with different servicechains. For example, RSVP-TE 99 may execute the Resource ReservationProtocol with Traffic Engineering extensions to exchange trafficengineering (TE) information, such as MPLS labels for enablinglabel-based packet forwarding. As another example, routing engine 86 mayuse GRE or IP-based tunneling protocols to established trafficengineered tunnels.

As described with respect to the techniques of this disclosure, routingengine 86 executes BGP 96 to establish BGP peer sessions with BGPspeakers and BGP listeners within the service provider network toexchange routing information. For example, routing engine 86 may use BGP96 to establish BGP peer sessions with other routers in the serviceprovider network and with a SDN controller of the service providernetwork, e.g., BGP peer session 18 with SDN controller 19 from FIG. 1 orBGP peer session 33 with SDN controller 32 from FIG. 2. Routing engine86 may then receive BGP route advertisements for a given service fromthe SDN controller via the BGP peer session. The advertised BGP routesrepresent unique tunnels established by the SDN controller that eachextends from router 80 to a respective one of a set of two or morevirtual service instances of the given service. In this way, router 80is able to automatically discover the virtual service instancesconfigured to apply a given service to packets.

In accordance with the techniques described in this disclosure, routingengine 86 executes a traffic load balancer (TLB) daemon 94 thatmaintains a load balancing group for the given service that includeseach of the BGP routes for the given service. In this way, each of thevirtual service endpoints of the BGP routes will be viewed as realservices and used as load balancing entities. The load balancing groupfor the given service may be maintained in a virtual routing andforwarding (VRF) table (not shown) for the given service within FIB 106.For example, TLB daemon 94 interacts with kernel 100 (e.g., by way ofAPI calls) to install the BGP routes in the VRF table for the givenservice in FIB 106. In some cases, the BGP routes stored in the VRFtable comprise an ECMP next hop route set that includes two or moremember routes to reach the two or more virtual service instances of thegiven service. TLB daemon 94 stores each of the BGP routes in the VRFtable with a tunnel identifier for the unique tunnel represented by therespective one of the BGP routes. The tunnel identifier for a particularunique tunnel includes a tunnel address of a GRE tunnel that extendsfrom router 80 to a physical network device, and a label assigned to oneof the virtual service instances of the given service running on thephysical network device.

Routing engine 86 communicates data representative of a software copy ofthe FIB 106 into each of PFEs 114 to control forwarding of trafficwithin the data plane. This allows the software FIB stored in memory(e.g., RAM) in each of PFEs 114 to be updated without degradingpacket-forwarding performance of router 80. In some instances, routingengine 86 may derive separate and different software FIBs for eachrespective PFEs 114. In addition, one or more of PFEs 114 include ASICs116 that PFEs 114 program with a hardware-copy of the FIB based on thesoftware FIBs (i.e., hardware versions of the software FIBs) copied toeach respective PFE 114.

For example, kernel 100 executes on master microprocessor 102 and maycomprise, for example, a UNIX operating system derivative such as Linuxor Berkeley Software Distribution (BSD). Kernel 100 processes kernelcalls from BGP 96, IGP 98, and RSVP-TE 99 to generate forwardinginformation in the form of FIB 106 based on the network topologyrepresented in RIB 104, i.e., performs route resolution and pathselection. Typically, kernel 100 generates FIB 106 in the form of radixor other lookup tree to map packet information (e.g., header informationhaving destination information and/or a label stack) to next hops andultimately to interface ports of interface cards associated withrespective PFEs 114. FIB 106 may associate, for example, networkdestinations with specific next hops and corresponding IFCs 112. Asdescribed above, FIB 106 may also include one or VRF tables (not shown)that each store BGP routes to virtual service instances of a givenservice.

Master microprocessor 102 executing on kernel 100 programs PFEs 114 toinstall copies of the FIB 106 including the VRF tables. Microprocessor102 may comprise one or more general- or special-purpose processors suchas a digital signal processor (DSP), an application specific integratedcircuit (ASIC), a field programmable gate array (FPGA), or any otherequivalent logic device. Accordingly, the terms “processor” or“controller,” as used herein, may refer to any one or more of theforegoing structures or any other structure operable to performtechniques described herein.

In the example illustrated in FIG. 4, ASICs 116 are microcode-controlledchipsets (i.e., forwarding circuits) programmably configured by a slavemicroprocessor executing on each of PFEs 114. When forwarding packets,control logic with each of ASICs 116 traverses the forwardinginformation (i.e., FIB 106) received from routing engine 86 and, uponreaching a FIB entry for the packet (e.g., a leaf node),microcode-implemented control logic automatically selects a forwardingnext hop and processes the packets in accordance with the operationsdefined within the next hop. In this way, ASICs 116 of PFEs 114 processpackets by performing a series of operations on each packet overrespective internal packet forwarding paths as the packets traverse theinternal architecture of router 80. Operations may be performed, forexample, on each packet based on any of a corresponding ingressinterface, an ingress PFE 114, an egress PFE 114, an egress interface orother components of router 80 to which the packet is directed prior toegress, such as one or more service physical interface cards (PICs)124A-124N (collectively, “service PICs 124”) of PFEs 114. PFEs 114 eachinclude forwarding structures that, when executed, examine the contentsof each packet (or another packet property, e.g., incoming interface)and on that basis make forwarding decisions, apply filters, and/orperform accounting, management, traffic analysis, and load balancing,for example. In accordance with the techniques of this disclosure,router 80 includes traffic load balancers (TLBs) 126A-126N (“TLBs 126”)running on service PICs 124 of PFEs 114. TLBs 126 may comprise statefulor stateless traffic load balancers.

In general, kernel 100 may generate FIB 106 and thereby program ASICs116 to store forwarding structures associated with each service chain.For example, ASICs 116 may be configured with forwarding informationthat specifies traffic engineering information, such as IP headerinformation or MPLS labels, as well as operations for causingprogrammable ASICs 116 to encapsulate subscriber packets in accordancewith the forwarding information. Service PICs 124 of PFEs 114 may besimilarly programed to store forwarding structures of the VRF tableswithin FIB 106 associated with a given service. For example, one or moreof service PICs 124 may be configured with a load balancing group for agiven service that includes BGP routes representing unique tunnels tovirtual service instances of the given service.

The techniques of this disclosure enable TLBs 126 within service PICs124 to apply a load balancing algorithm in order to select one of theBGP routes from the load balancing group for the given service on whichto forward subscriber packets assigned to the given service. Uponselecting one of the BGP routes from the load balancing group, ASICs 116may encapsulate the received packet with the tunnel identifier for theunique tunnel that extends to the virtual service instance representedby the selected one of the BGP routes. In this way, ASICs 116 mayprocess subscriber packets to select particular service paths for eachpacket and encapsulate the subscriber packets in accordance with theselected service or service chain.

As a specific example of the disclosed techniques, TLB daemon 94executing in routing engine 86 of router 34 may create real services foreach of the virtual service endpoints of the BGP routes. TLB daemon 94may create the real services with tunnel identifiers pointing to theunique tunnels represented by the BGP ECNIP next hop route sets, andassociate the tunnel identifiers with a wildcard virtual server. Thewildcard virtual server may be listening for traffic on one of servicePICs 124, e.g., service PIC 124A. In this case, PFEs 114 may sendtraffic to the service PIC 124A based on configured filters. The servicePIC 124A identifies the traffic arriving on this wildcard virtualserver, and TLB 126A on service PIC 124A does load balancing based onconfigured load balancing algorithms. The service PIC 124A may thensetup sessions to link the forward and reverse flow for subsequentpackets.

When the next update on this ECMP next hop route set arrives at routingengine 86 from the SDN controller via BGP 96, TLB daemon 94 may checkfor any changes in the members of the next hops in the VRF. If there isany addition of real services, TLB daemon 94 adds the real services tothe load balancing group in the VRF. If an ECMP next hop route setmember is missing, TLB daemon 94 deletes the entry from the loadbalancing group in the VRF. For the real services that are deleted fromthe ECMP next hop route set, TLB 126A treats it as a deletion or downevent on that real service (i.e., the VM or virtual entity hosting thevirtual service instance). For transmission control protocol (TCP)clients, TCP Reset may be sent on the unique tunnels established to thereal services. For user datagram protocol (UDP) clients, the session maybe closed with a predefined timeout.

To support load balancing in a SDN environment, TLBs 126 in service PICs124 may require that data packets be routed by a different address thanthe destination address of the packet. In such cases, TLBs 126 inservice PICs 124 may provide the real service address (i.e., the tunnelidentifier to route the packet to the given real service instance) inthe JavaScript File Manager (JFM) Cookie along with the data packet. ThePFEs 114 may then route the packet based on the tunnel identifier in theJFM Cookie instead of the destination IP address. Further details of agateway network device capable of routing packets of packet flows basedon cookies are described in U.S. patent application Ser. No. 14/042,685,filed Sep. 30, 2013, and U.S. patent application Ser. No. 14/228,706,filed Mar. 28, 2014, the entire contents of each of which areincorporated herein by reference.

The architecture of router 80 illustrated in FIG. 5 is shown for examplepurposes only. This disclosure is not limited to this architecture. Inother examples, router 80 may be configured in a variety of ways. In oneexample, some of the functionally of control unit 82 may be distributedwithin IFCs 112. Control unit 82 may be implemented solely in software,or hardware, or may be implemented as a combination of software,hardware, or firmware. For example, control unit 82 may comprise one ormore of a processor, a programmable processor, a general purposeprocessor, an integrated circuit, an Application Specific IntegratedCircuit (ASIC), a Field Programmable Gate Array (FPGA), or any type ofhardware unit capable of implementing the techniques described herein.Control unit 82 may further include one or more processors that executesoftware instructions stored on a computer readable storage medium, suchas random access memory (RAM), read only memory (ROM), programmable readonly memory (PROM), erasable programmable read only memory (EPROM),electronically erasable programmable read only memory (EEPROM),non-volatile random access memory (NVRAM), flash memory, a hard disk, aCD-ROM, a floppy disk, a cassette, magnetic media, optical media, orother computer-readable storage media. In some instances, thecomputer-readable storage medium may include instructions that cause aprogrammable processor to perform the techniques described herein.

FIG. 5 is a block diagram illustrating an example server 130 thatprovides an operating environment for one or more service nodes144A-144M (collectively “service nodes 144”). In some examples, server130 may comprise a server within a data center, such as services complex9 from FIG. 1.

In this example, server 130 includes a network interface 136 to receivetunnel packets 132 over a plurality of unique tunnels 134A-134N(collectively “tunnels 134”). Each of the unique tunnels 134 correspondsto a different one of a plurality of services, which may be front-endservices in different service chains that comprise different orderedsets of one or more stateful network services to be applied to packetflows. Each of the tunnel packets 132 encapsulates a subscriber packetand includes a tunnel identifier for one of the unique tunnels 134 thatextends to one of the virtual service instances, e.g., service nodes144A-144M (collectively “service nodes 144”), configured on servicer 130to apply a given service to the tunnel packets 132. As described above,the tunnel identifier for one of the unique tunnels 134, e.g., uniquetunnel 134A, that extends to one of the virtual service instances, e.g.,service node 144A, includes a tunnel address of a GRE tunnel to reachserver 130 with a label assigned to service node 144A.

In the example of FIG. 5, server 130 includes a microprocessor 138executing hypervisor 140 to provide an execution environment for avirtual router 142 and one or more virtual machines 146A-146M(collectively “virtual machines 146”) within service nodes 144. Virtualmachines 146 of service nodes 144 provide virtual service endpoints forunique tunnels 134. Each of the virtual machines 146 executes networkservices software, such as firewall instance 150 within service node144A and HTTP filter instance 152 within service node 144M, to apply oneor more of the network services to the packet flows.

Upon receipt of tunnel packets 132 via one of unique tunnels 134, e.g.,unique tunnel 134A, virtual router 142 processes and removes the tunnelidentifiers of each of the tunnel packets to determine which of servicenodes 144 to send the respective packet. For example, for a specifictunnel packet, virtual router 142 first removes the GRE tunnel addressused to reach server 130 from the tunnel packet, and then reads andremoves the label assigned to the service instance endpoint of uniquetunnel 134A. Based on the label, virtual router 142 forwards thedecapsulated packet onto the appropriate tap interface of one of servicenodes 144, e.g., service node 144A. In this example, virtual machine146A of service node 144A receives the decapsulated packet and appliesthe service of firewall instance 150 to the decapsulated packet.

FIG. 6 is a flowchart illustrating exemplary operation of a networkdevice, e.g., a router or gateway, and a SDN controller, in accordancewith the techniques described herein. The operation illustrated in FIG.6 is described herein with respect to SDN controller 32 and router 34from FIG. 2. In other examples, the operation illustrated in FIG. 6 maybe performed by SDN controller 19 and gateway 8 from FIG. 1, or SDNcontroller 60 from FIG. 3 and router 80 from FIG. 4.

Router 34 establishes BGP peer session 33 with SDN controller 32 (202),and SDN controller 62 also establishes BGP peer session 33 with router34 (204). In some examples, SDN controller 32 may initiate the BGPpeering with router 34. In other examples, router 34 may initiate theBGP peering with SDN controller 32.

SDN controller 32 may also establish a set of two or more virtualservice instances configured to apply a given service to a packet. Insome cases, SDN controller 32 may establish the virtual serviceinstances as two or more VMs running on a physical network device, suchas a server. For example, as illustrated in FIG. 2, SDN controller 32may establish the first virtual service instance of VM-S 11 44 and thesecond virtual service instance of VM-S12 45 on server-1 40 for thegreen service. In other cases, SDN controller 32 may establish thevirtual service instances as at least one VM running on a first physicalnetwork device and at least one VM running on a second physical networkdevice. For example, as illustrated in FIG. 2, SDN controller 32 mayestablish the first and second virtual service instances as VM-S11 44and VM-S12 45 on server-1 40, and establish the third virtual serviceinstance of VM-S21 46 on server-2 42 for the green service.

SDN controller 32 establishes unique tunnels that each extends fromrouter 34 to a respective one of the virtual service instancesconfigured to apply the given service to the packet (206). Asillustrated in FIG. 2, for the green service, SDN controller 32establishes a first unique tunnel that extends from router 34 to VM-S 1144 hosted on server-1 40, the first unique tunnel being identified by atunnel address of GRE tunnel 50 with label L101 assigned to the firstvirtual service instance of VM-S11 44. SDN controller 32 establishes asecond unique tunnel that extends from router 34 to VM-S12 45 hosted onserver-1 40, the second unique tunnel being identified by a tunneladdress of GRE tunnel 50 with label L102 assigned to the second virtualservice instance of VM-S12 45. SDN controller 32 also establishes athird unique tunnel that extends from router 34 to VM-S21 46 hosted onserver-2 42, the third unique tunnel being identified by a tunneladdress of GRE tunnel 52 with label L103 assigned to the third virtualservice instance of VM-S21 46.

SDN controller 32 then calculates BGP routes that represent the uniquetunnels that each extends to the respective one of the set of virtualservice instances for the given service (208). SDN controller 32advertises the BGP routes for the given service to router 34 via BGPpeer session 33 as load balancing entities (210). In some cases, SDNcontroller 32 calculates and advertises an ECMP next hop route set thatincludes two or more member routes to reach the two or more virtualservice instances configured to apply the given service. For example,SDN controller 32 may calculate and advertise an ECMP next hop route setfor the green service including a first member route to reach the firstvirtual service instance of VM-S11 44 via the first unique tunnel, asecond member route to reach the second virtual service instance ofVM-S12 45, and a third member route to reach the third virtual serviceinstance of VM-S21 46.

Router 34 receives the BGP route advertisements for the given service asload balancing entities from SDN controller 32 on BGP peer session 33(212). Router 34 maintains a load balancing group for the given servicethat includes each of the BGP routes for the given service (214). Insome cases, router 34 receives an ECMP next hop route set from SDNcontroller 32 that includes two or more member routes to reach the twoor more virtual service instances configured to apply the given service.For example, router 34 may update the load balancing group for the greenservice based on the ECMP next hop route set to include a first memberroute to reach the first virtual service instance of VM-S11 44 via thefirst unique tunnel, a second member route to reach the second virtualservice instance of VM-S12 45, and a third member route to reach thethird virtual service instance of VM-S21 46.

In one example, router 34 maintains the load balancing group in a VRFtable for the given service. In this example, for each of the BGP routesincluded in the load balancing group, the VRF table includes a tunnelidentifier for the unique tunnel that extends to the virtual serviceinstance represented by the respective one of the BGP routes. Asillustrated in FIG. 2, router 34 may maintain the load balancing groupfor the green service in VRF-green 36. For example, VRF-green 36 mayinclude a first tunnel identifier for the first unique tunnel thatextends to the first virtual service instance of VM-S11 44 comprising atunnel address of GRE tunnel 50 with label L101 assigned to VM-S11 44.VRF-green 36 may also include a second tunnel identifier for the secondunique tunnel that extends to the second virtual service instance ofVM-S12 45 comprising a tunnel address of GRE tunnel 50 with label L102assigned to VM-S12 45. VRF-green 36 may further include a third tunnelidentifier for the third unique tunnel that extends to the third virtualservice instance of VM-S21 46 comprising a tunnel address of GRE tunnel52 with label L103 assigned to VM-S21 46.

SDN controller 32 may continue to advertise BGP route updates for thegiven service to router 34 via BGP peer session 33 whenever changes tothe BGP routes occur. As one example, the BGP route updates may includenotifications of new virtual service instances established on one ofserver-1 40, server-2 42, or another server, for the given service. Asanother example, the BGP route updates may include notifications ofremoved virtual service instances from one of server-1 40, server-2 42,or another server, for the given service.

Router 34 may receive the BGP route update advertisements from SDNcontroller 32 on BGP peer session 33. Based on the received BGP routeupdates, router 34 updates the load balancing group for the givenservice. In one example, router 34 may add BGP routes to the loadbalancing group for any new virtual service instance notifications inthe BGP route updates. In another example, router 34 may delete BGProutes from the load balancing group for any removed virtual serviceinstance notifications in the BGP route updates.

Router 34 receives a packet sourced from a first network device anddestined for a second network device (216). Upon receipt of the packet,router 34 determines one or more, or a chain, of services to apply tothe packet prior to forwarding the packet toward its destination of thesecond network device. In accordance with the techniques of thisdisclosure, after determining to apply the given service to the receivedpacket, router 34 selects, according to a load balancing algorithm, oneof the BGP routes from the load balancing group on which to forward thepacket to the one of the virtual service instances for application ofthe given service (218). For example, in the case where router 34determines to apply the green service to the received packet, router 34selects one of the BGP routes from the load balancing group included inVRF-green 36 on which to forward the packet to one of the virtualservice instances of VM-S11 44, VM-S12 45, or VM-S21 46 configured toapply the green service to the packet.

Upon selecting one of the BGP routes from the load balancing group,router 34 may encapsulate the received packet with the tunnel identifierfor the unique tunnel that extends to the virtual service instancerepresented by the selected one of the BGP routes. For example, in thecase where router 34 selects the BGP route that represents the secondunique tunnel that extends to the second virtual service instance ofVM-S12 45, router 34 may encapsulate the received packet with the secondtunnel identifier that includes a tunnel address of GRE tunnel 50 withlabel L102 assigned to VM-S12 45. Router 34 then forwards theencapsulated packet according to the tunnel identifier for the selectedBGP route to the virtual service instance for application of the givenservice to the packet (220).

The techniques described herein may be implemented in hardware,software, firmware, or any combination thereof. Various featuresdescribed as modules, units or components may be implemented together inan integrated logic device or separately as discrete but interoperablelogic devices or other hardware devices. In some cases, various featuresof electronic circuitry may be implemented as one or more integratedcircuit devices, such as an integrated circuit chip or chipset.

If implemented in hardware, this disclosure may be directed to anapparatus such as a processor or an integrated circuit device, such asan integrated circuit chip or chipset. Alternatively, or additionally,if implemented in software or firmware, the techniques may be realizedat least in part by a computer-readable data storage medium comprisinginstructions that, when executed, cause a processor to perform one ormore of the methods described above. For example, the computer-readabledata storage medium may store such instructions for execution by aprocessor.

A computer-readable medium may form part of a computer program product,which may include packaging materials. A computer-readable medium maycomprise a computer data storage medium such as random access memory(RAM), read-only memory (ROM), non-volatile random access memory(NVRAM), electrically erasable programmable read-only memory (EEPROM),Flash memory, magnetic or optical data storage media, and the like. Insome examples, an article of manufacture may comprise one or morecomputer-readable storage media.

In some examples, the computer-readable storage media may comprisenon-transitory media. The term “non-transitory” may indicate that thestorage medium is not embodied in a carrier wave or a propagated signal.In certain examples, a non-transitory storage medium may store data thatcan, over time, change (e.g., in RAM or cache).

The code or instructions may be software and/or firmware executed byprocessing circuitry including one or more processors, such as one ormore digital signal processors (DSPs), general purpose microprocessors,application-specific integrated circuits (ASICs), field-programmablegate arrays (FPGAs), or other equivalent integrated or discrete logiccircuitry. Accordingly, the term “processor,” as used herein may referto any of the foregoing structure or any other structure suitable forimplementation of the techniques described herein. In addition, in someaspects, functionality described in this disclosure may be providedwithin software modules or hardware modules.

Various embodiments have been described. These and other embodiments arewithin the scope of the following examples.

What is claimed is:
 1. A method comprising: establishing, by asoftware-defined networking (SDN) controller, a border gateway protocol(BGP) peer session with a gateway network device; establishing, by theSDN controller, two or more unique tunnels that each extends from thegateway network device to a respective one of a set of two or morevirtual service instances configured to apply a given service to apacket; calculating, by the SDN controller, BGP routes for the givenservice, wherein each of the BGP routes represents one of the uniquetunnels that extends from the gateway network device to the respectiveone of the virtual service instances; and advertising, by the SDNcontroller and to the gateway network device via the BGP peer session,the BGP routes for the given service as load balancing entities.
 2. Themethod of claim 1, wherein calculating the BGP routes for the givenservice comprises calculating an equal cost multipath (ECMP) next hoproute set having two or more member routes to reach the two or morevirtual service instances configured to apply the given service.
 3. Themethod of claim 1, wherein the one of the unique tunnels that extendsfrom the gateway network device to the respective one of the virtualservice instances is identified by a tunnel address of a generic routingencapsulation (GRE) tunnel with a label assigned to the respective oneof the virtual service instances.
 4. The method of claim 1, furthercomprising advertising, by the SDN controller and to the gateway networkdevice via the BGP peer session, BGP route updates for the givenservice, wherein the BGP route updates include notifications of newvirtual service instances for the given service or notifications ofremoved virtual service instances for the given service.
 5. The methodof claim 1, further comprising establishing, by the SDN controller, thetwo or more virtual service instances as two or more virtual machinesrunning on a physical network device.
 6. The method of claim 1, furthercomprising establishing, by the SDN controller, the two or more virtualservice instances as at least one first virtual machine running on afirst physical network device and at least one second virtual machinerunning on a second physical network device.
 7. A method comprising:establishing, by a gateway network device, a border gateway protocol(BGP) peer session with a software-defined networking (SDN) controller;receiving, by the gateway network device and from the SDN controller viathe BGP peer session, advertisements of BGP routes for a given serviceas load balancing entities, wherein each of the BGP routes representsone of a set of two or more unique tunnels that each extends from thegateway network device to a respective one of a set of two or morevirtual service instances configured to apply the given service to apacket; maintaining, by the gateway network device, a load balancinggroup for the given service that includes each of the BGP routes for thegiven service; and selecting, by the gateway network device andaccording to a load balancing algorithm, one of the BGP routes from theload balancing group on which to forward the packet to the respectiveone of the virtual service instances for application of the givenservice.
 8. The method of claim 7, wherein the BGP routes for the givenservice comprise an equal cost multipath (ECMP) next hop route sethaving two or more member routes to reach the two or more virtualservice instances configured to apply the given service.
 9. The methodof claim 7, wherein maintaining the load balancing group for the givenservice comprises maintaining the load balancing group in a virtualrouting and forwarding (VRF) table for the given service, wherein, foreach of the BGP routes, the VRF table includes a tunnel identifier forthe one of the unique tunnels that extends to the respective one of thevirtual service instances represented by the respective one of the BGProutes.
 10. The method of claim 9, wherein the tunnel identifier for theone of the unique tunnels that extends to the respective one of thevirtual service instances comprises a tunnel address of a genericrouting encapsulation (GRE) tunnel with a label assigned to therespective one of the virtual service instances.
 11. The method of claim9, further comprising: receiving the packet sourced from a first networkdevice and destined for a second network device; upon selecting the oneof the BGP routes from the load balancing group, encapsulating thepacket with the tunnel identifier for the one of the unique tunnels thatextends to the respective one of the virtual service instancesrepresented by the selected one of the BGP routes; and forwarding theencapsulated packet based on the tunnel identifier to the one of thevirtual service instances for application of the given service to thepacket.
 12. The method of claim 7, further comprising receiving, by thegateway network device and from the SDN controller via the BGP peersession, advertisements of BGP route updates including at least one ofnotifications of new virtual service instances for the given service ornotifications of removed virtual service instances for the givenservice.
 13. The method of claim 12, further comprising updating theload balancing group for the given service based on the BGP routeupdates, wherein updating the load balancing group includes at least oneof adding BGP routes to the load balancing group for any new virtualservice instances or deleting BGP routes from the load balancing groupfor any removed virtual service instances.
 14. A software-definednetworking (SDN) controller comprising: a plurality of networkinterfaces; and a control unit comprising a processor executing a bordergateway protocol (BGP) to establish a BGP peer session on one of thenetwork interfaces with a gateway network device, and establish two ormore unique tunnels that each extends from the gateway network device toa respective one of a set of two or more virtual service instancesconfigured to apply a given service to a packet, the processorconfigured to calculate BGP routes for the given service, wherein eachof the BGP routes represents one of the unique tunnels that extends fromthe gateway network device to the respective one of the virtual serviceinstances, and output the BGP routes for the given service to the one ofthe network interfaces for advertising as load balancing entities to thegateway network device via the BGP peer session.
 15. The SDN controllerof claim 14, wherein, to calculate the BGP routes for the given service,the processor is configured to calculate an equal cost multipath (ECMP)next hop route set having two or more member routes to reach the two ormore virtual service instances configured to apply the given service.16. The SDN controller of claim 14, wherein the one of the uniquetunnels that extends from the gateway network device to the respectiveone of the virtual service instances is identified by a tunnel addressof a generic routing encapsulation (GRE) tunnel with a label assigned tothe respective one of the virtual service instances.
 17. The SDNcontroller of claim 14, wherein the processor is configured to outputBGP route updates for the given service to the one of the networkinterfaces for advertising to the gateway network device via the BGPpeer session, wherein the BGP route updates include notifications of newvirtual service instances for the given service or notifications ofremoved virtual service instances for the given service.
 18. The SDNcontroller of claim 14, wherein the processor is configured to establishthe two or more virtual service instances as two or more virtualmachines running on a physical network device.
 19. The SDN controller ofclaim 14, wherein the processor is configured to establish the two ormore virtual service instances as at least one first virtual machinerunning on a first physical network device and at least one secondvirtual machine running on a second physical network device.
 20. Agateway network device comprising: a plurality of network interfaces; acontrol unit comprising a processor executing a boarder gateway protocol(BGP) to establish a BGP peer session on one of the network interfaceswith a software-defined networking (SDN) controller, the processorconfigured to receive advertisements of BGP routes for a given serviceas load balancing entities on the one of the network interfaces via theBGP peer session with the SDN controller, wherein each of the BGP routesrepresents one of a set of two or more unique tunnels that each extendsfrom the gateway network device to a respective one of a set of two ormore virtual service instances configured to apply the given service toa packet, and maintain a load balancing group for the given service thatincludes each of the BGP routes for the given service; and a forwardingcircuit programmed with forwarding information associated with the BGProutes, the forwarding circuit configured to select one of the BGProutes for the packet from the load balancing group according to a loadbalancing algorithm, wherein the forwarding circuit outputs the packetto one of the network interfaces for forwarding to the respective one ofthe virtual service instances for application of the given service. 21.The gateway network device of claim 20, wherein the BGP routes for thegiven service comprise an equal cost multipath (ECMP) next hop route sethaving two or more member routes to reach the two or more virtualservice instances configured to apply the given service.
 22. The gatewaynetwork device of claim 20, wherein the processor is configured tomaintain a virtual routing and forwarding (VRF) table for the givenservice based on the BGP routes for the given service, wherein, for eachof the BGP routes, the VRF table includes a tunnel identifier for therespective one of the unique tunnels that extends to the respective oneof the virtual service instances represented by the respective one ofthe BGP routes.
 23. The gateway network device of claim 22, wherein thetunnel identifier for the one of the unique tunnels that extends to therespective one of the virtual service instances is identified by atunnel address of a generic routing encapsulation (GRE) tunnel with alabel assigned to the respective one of the two or more virtual serviceinstances.
 24. The gateway network device of claim 22, wherein theforwarding circuit: receives the packet sourced from a first networkdevice and destined for a second network device; upon selection of theone of the BGP routes from the load balancing group, encapsulates thepacket with the tunnel identifier for the one of the unique tunnels thatextends to the respective one of the virtual service instancesrepresented by the selected one of the BGP routes; and outputs theencapsulated packet to the one of the network interfaces for forwardingto the respective one of the virtual service instances based on thetunnel identifier for application of the given service to the packet.25. The gateway network device of claim 20, wherein the processor isconfigured to receive advertisements of BGP route updates on the one ofthe network interfaces via the BGP peer session with the SDN controller,wherein the BGP route updates include at least one of notifications ofnew virtual service instances for the given service or notifications ofremoved virtual service instances for the given service.
 26. The gatewaynetwork device of claim 25, wherein the processor is configured toupdate the load balancing group for the given service based on the BGProute updates, the processor configured to at least one of add BGProutes to the load balancing group for any new virtual service instancesor delete BGP routes from the load balancing group for any removedvirtual service instances.